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DETAILED ACTION 



1. 



This action is in response to the amendment filed on 6/6/2008. 



2. 



Claims 1-10, 12-14, and 16-20 are currently pending. 



Claim Objections 



3. Claim12 is objected to because of the following informalities: claim 12 has been 
amended to be dependent upon its self. For purposes of examination, claim 12 will be 
read as being dependent upon claim 1 . Appropriate correction is required. 

Claim Rejections - 35 USC § 101 

4. The previous rejections of Claims 1-7 and 11-12 under 35 U.S.C. 101 are 
hereby withdrawn due to the 6/6/2008 amendment. 



5. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 



6. Claims 1, 2, 4-6, 8-10, 12-14, 16, and 18-20 are rejected under 35 U.S.C. 103(a) 
as being unpatentable over Tomm (US 6,246,403) in view of Tran (US 6,054,990), and 



Claim Rejections - 35 USC § 103 



further in view of Scott et al. (US 5,675,752), hereinafter Scott. 
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7. As per claims 1,13, and 20, Tomm teaches the following: 

central processing unit and at least a memory, including a computer program for 
instructing the processing unit, (column 3, lines 44-46), i.e. Fig. 1 is a pictorial 
representation of a computer 4 used in accordance with the present invention by a user 
to create a text file, to represent: 

the sketch being representative of a graphic user interface (GUI) to be created, 
(column 1, lines 41-43), i.e. drawing programs are currently available on the market 
which enable a person to create a GUI by drawing the desired GUI; 

a hierarchy determiner for determining, from the conformed versions, a hierarchy 
of relative containment among said conformed versions, (column 7, lines 9-1 1 ), i.e. 
once the list of boxes has been generated, the collection of boxes is organized into a 
data tree structure , as indicated by block 70; and 

a description generator for generating, from said hierarchy, a user interface 
description for creating the GUI, (abstract), once the data tree structure has been 
created, a GUI generating software program can use the data tree structure to generate 
the GUI . 

said hierarchy reflecting a stacking order based on at least one of an annotation 
to a sketched version of an objec t a containment of un-annotated objects within an 
object and data types of un-annotated objects, wherein said annotation indicates a 
priority for the annotated object and said un-annotated objects are recognized to obtain 
a priory of the containing object; and wherein said stacking order determines to which of 
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two objects has priority to overlay the other of the two in said GUI, (column 4, lines 43- 
45), i.e. it will be assumed herein that if a point lies on the top or right edges 1 7 and 1 9 
of the box 10, the box 10 encloses the point. 

However, Tomm does not explicitly teach the inner workings of using a sketch 
recognition method for input. Tran teaches the following: 

a sketch identifier for examining a manual sketch of objects to identify 
sketched versions of the objects, (abstract), i.e. an object classification and recognition 
capability is provided to allow the present invention to recognize objects being sketched 
by the user; 

a sketch normalizer for conforming the identified sketched versions to resemble 
respective reference images, (abstract), i.e. an object classification and recognition 
capability is provided to allow the present invention to recognize objects being sketched 
by the user and convert the hand sketches into computer aided design like drawings; 

It would have been obvious to one of ordinary skill in the art to have 
modified Tomm's GUI design method with the sketch recognition method of Tran. One 
of ordinary skill in the art would have been motivated to have made such modifications 
because Tomm directly teaches in their background of invention in column 1 , lines 41- 
57, that at the time it was well known in the art that their existed GUI creating programs 
which accepted user input through a drawing program. Furthermore, as Tomm teaches 
in their abstract, their text file which they use to generate a GUI comprises a "two- 
dimensional pictorial representation". As was well known in the art, CAD drawings, of 
which Tran is directed towards, may be represented as a two-dimensional pictorial 



Application/Control Number: 10/575,575 Page 5 

Art Unit: 2176 

representation. Still further, as Tomm teaches in their abstract, their method generates 
a GUI from a text file. Tran teaches in column 7, line 66 - column 8, line 1 , that as 
"drawn" data is sent to their handwriting recognizer, the written instructions are 
converted into text data. 

8. Regarding claim 2, modified Tomm teaches the method of claim 1 as described 
above. Tran further teaches the following: 

said reference images comprise vector images, (column 14, lines 21-23), i.e. 
from step 153 to 155, the routine next determines whether the drawing is to be saved as 
a hand-drawn bitmap sketch or as a converted series of vectors and/or predefined 
objects in step 156. 

9. Regarding claim 4, modified Tomm teaches the method of claim 1 as described 
above. Tran further teaches the following: 

the manual sketch includes characters, and wherein the sketch identifier is 
configured for applying optical character recognition (OCR), (column 8, lines 5-7), i.e. 
with suitable recognition software, text and numeric information can be entered into the 
pen-based computer system in a similar fashion. 



10. Regarding claim 5, modified Tomm teaches the method of claim 1 as described 
above. Tomm further teaches the following: 
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said description generator is further configured for generating the user interface 
description to contain a layout of said conformed versions, (abstract), i.e. the GUI 
constructing program scans the text file and determines the x and v coordinates of the 
corners of all of the boxes and the data type associated with each box. 

1 1 . Regarding claim 6, modified Tomm teaches the method of claim 1 as described 
above. Tomm further teaches the following: 

the description generator is configured to generate the user interface description 
into a format specific to a target platform for the GUI, (column 3, lines 37-43), i.e. it 
should also be noted that the GUI constructing program of the present invention is 
suitable for use with different types of windowing systems . 

12. Regarding claim 8, modified Tomm teaches the method of claim 1 as described 
above. Tran further teaches the following: 

an electronic storage medium, (column 6, lines 57-59), i.e. the processor 20 is 
also connected to a random access memory 22 for storing various run-time variables 
and data arrays; 

a hand-held pen for creating the sketch, (column 7, lines 31-35), i.e. in one 
embodiment where the computer accepts handwritings as an input medium from the 
user, a combination pen/ digitizer unit 25, and a display LCD panel 26 having a viewing 
screen exposed along one of the planar sides of the enclosure are provided; and 
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a digitizer for recording into the medium the sketch in real time as the sketch is 
being created, (column 7, lines 31-35), i.e. in one embodiment where the computer 
accepts handwritings as an input medium from the user, a combination pen /digitizer unit 
25, and a display LCD panel 26 having a viewing screen exposed along one of the 
planar sides of the enclosure are provided. 

13. Regarding claim 9, modified Tomm teaches the method of claim 8 as described 
above. Tomm further teaches the following: 

the apparatus stores in said medium a normalized sketch comprising the 
conformed versions, (column 3, lines 64-66), i.e. once the text file has been created, the 
text file is stored in a memory device which may be comprised by computer 4. 

However, as described above, Tomm does not explicitly describe the inner 
workings of a sketch recognition method for input. Tran further teaches the following: 

said apparatus further comprising a sketch editor for editing said normalized 
sketch stored in said medium, (column 11, lines 34-37), i.e. in Fig. 2, the menu 100 
allows the user to control the sketching system . Preferably, the menu 100 is a pull- 
down menu with a number of choices, including File, Edit , Object and View; 

said digitizer being configured for augmenting, according to input from the pen, 
said normalized sketch stored in said medium, (column 12, lines 32-36), i.e. referring 
now to Fig. 3, the operation of the magnifier 108 of Fig. 2 is disclosed. Upon beginning 
the process in step 120 of Fig. 2, the routine displays a magnifier icon whenever the pen 
touches the LCD screen 26 of the computer system. 
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14. Regarding claim 10, modified Tomm teaches the method of claim 1 as described 
above. Tran further teaches the following: 

an electronic storage medium for storing said reference images, (column 7, lines 
2-3), i.e. the RAM 22 stores a database of the drawings of the present invention; and 

wherein the sketch identifier is configured for using the stored reference images 
in identifying said sketched versions, (column 13, lines 44-49), i.e. the routine of Fig. 5 
will compare the sketches against a library containing definitions and symbols for 
predetermined objects if the user does not specify in advance that her or she is writing 
text, in such case the handwriting recognition engine is to be applied. 

15. Regarding claim 12, modified Tomm teaches the method of claim 1 as described 
above. Tomm further teaches the following: 

said apparatus being further configured to recognize that said annotation 
indicates priority based on a dividing line within said sketched version of an object, 
(column 4, lines 43-45), i.e. it will be assumed herein that if a point lies on the top or 
right edges 1 7 and 1 9 of the box 1 0, the box 1 0 encloses the point, (column 4, line 67 - 
column 5, lines 5), i.e. the boxes 23 and 25 are dialog boxes containing GUI 
components commonly referred to as push buttons. T he indication "button" on the top 
edges of the boxes 23 and 25 corresponds to the data type of the boxes and indicates 
to the GUI constructing program that the box data type is push button. 
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The examiner interprets Tomm's teaching of a top edge line which visually 
defines a box to encompass applicant's limitation of a "dividing line". 

16. Regarding claim 14, modified Tomm teaches the method of claim 13 as 
described above. Tran further teaches the following: 

the sketching step further includes the step of sketching, as an annotation to at 
least one of the objects, a label of a function of the object in said GUI, (column 4, lines 
49-57), i.e. each box also has a name and a data type associated with it , which are text 
strings that are used by the windowing system executing the GUI construction program 
to create windows and other GUI components. The name is simply a word or a 
combination of words that will be displayed in the GUI window associated with the box. 
The data type informs the GUI constructing program of the manner in which the box 
must be handled by the GUI constructing program. 

1 7. Regarding claim 16, modified Tomm teaches the method of claim 1 3 as 
described above. Tran further teaches the following: 

at least one of the sketched versions of an object intersects another sketched 
version of an object, (column 6, lines 1-4), i.e. this data tree structure contains the 
relative dimensions and locations of the boxes that appear in the input text file, the 
names and types of the boxes, and the enclosure relationships among the boxes : and 

wherein the sketching step further includes the step of sketching, as an 
annotation to at least one of two mutually intersecting ones of the versions, a label of a 
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function of the respective object in said GUI, (column 4, lines 49-57), i.e. each box also 
has a name and a data type associated with it , which are text strings that are used by 
the windowing system executing the GUI construction program to create windows and 
other GUI components. The name is simply a word or a combination of words that will 
be displayed in the GUI window associated with the box. The data type informs the GUI 
constructing program of the manner in which the box must be handled by the GUI 
constructing program. 

18. Regarding claim 18, modified Tomm teaches the method of claim 13 as 
described above. Tran further teaches the following: 

the sketching step further comprises the steps of: 

manipulating a pen by hand to create the sketch, (abstract), i.e. an object 
classification and recognition capability is provided to allow the present invention to 
recognize objects being sketched bv the user : and 

recording into the medium the sketch in real time as the sketch is being created, 
(column 7, lines 60-63), i.e. as the CPU 20 senses the position and movement of the 
stylus, it generates a corresponding image on the screen to create the illusion that the 
pen or stylus is drawing the image directly upon the screen . 



1 9. Regarding claim 1 9, modified Tomm teaches the method of claim 1 3 as 
described above. Tran further teaches the following: 
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the step of pre-storing said reference images to aid in the identification performed 
in the examining step, (column 13, lines 44-49), i.e. the routine of Fig. 5 will compare 
the sketches against a library containing definitions and symbols for predetermined 
objects if the user does not specify in advance that her or she is writing text, in such 
case the handwriting recognition engine is to be applied. 

20. Claim 3 is rejected under 35 U.S.C. 103(a) as being unpatentable over modified 
Tomm as applied to claim 1 above, and further in view of Eller et al. (US 5,287,417), 
hereinafter Eller. 

21 . Regarding claim 3, modified Tomm teaches the method of claim 1 as described 
above. However, neither Tomm nor Tran explicitly teaches refining the sketches. Eller 
teaches the following: 

the sketch normalizer is configured for straightening lines and making lines 
mutually parallel, (column 3, lines 59-61), i.e. after the objects are recognized, they are 
displayed with perfected angles and lines . 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have further modified the sketch recognition method of Tomm in 
view of Tran, with the refining method of Eller. One of ordinary skill in the art would 
have been motivated to have made such further modifications because Tomm directly 
teaches in their background of invention in column 1 , lines 41-57, that at the time it was 
well known in the art that their existed GUI creating programs which accepted user input 
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through a drawing program. Furthermore, both Tran and Eller are analogous art in the 
field of interpreting hand drawn input. 

22. Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over modified 
Tomm as applied to claim 1 above, and further in view of Carroll (US 2002/0085020). 

23. Regarding claim 7, modified Tomm teaches the method of claim 1 as described 
above. However, neither Tomm nor Tran explicitly teaches the description is written in a 
structured mark-up language. Carroll teaches the following: 

the description generator is configured for generating the description into a 
hierarchical, structured mark-up language, (abstract), i.e. an applications graphical user 
interface is specified using an XML document as an application interface file . At 
application compile time this application interface file is parsed, and the specifications 
therein used to retrieve graphical screen components from an interface library to create 
the user interface. 

It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have further modified the user interface document generation 
method of Tomm with the XML user interface document of Carroll. One of ordinary skill 
in the art would have been motivated to have made such modifications because both 
Tomm and Carroll are analogous art in the field of generating user interfaces from 
document files. 
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24. Claim 17 is rejected under 35 U.S.C. 103(a) as being unpatentable over modified 
Tomm as applied to claims 1 and 16 above, and further in view of Alexander (US 
6,118,451). 

25. Regarding claim 17, modified Tomm teaches the method of claim 16 as 
described above. Tomm teaches the following: 

the hierarchy determining step relatively positions in said hierarchy respective 
objects of said two mutually intersecting ones, (column 6, lines 1-4), i.e. this data tree 
structure contains the relative dimensions and locations of the boxes that appear in the 
input text file, the names and types of the boxes, and the enclosure relationships among 
boxes. 

As was well known in the art at the time, in a data tree structure, child nodes are 
connected to their parents, making them relatively positioned in the hierarchy. 

However, Tomm does not explicitly teach a method of relatively positioning 
boxes in the hierarchy based on an annotation . Alexander teaches the following: 

based on an annotation created in the annotation sketching step, (column 24, 
lines 53-57), i.e. the Z order list data structure 612 contains an identification 722 of all 
dialogs currently being managed by the dialog manager 402. Associated with each is a 
Z number 724 identifying the position in the visual sequence on the display screen. 

As Tomm's method already bases their hierarchical tree on "enclosure relations", 
e.g. overlays, it would have been obvious to one of ordinary skill in the art to have used 
the Z values of modified Tomm to position boxes in their hierarchical tree. 
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It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to have modified the indications of Tomm to further include the "Z 
number" value of Alexander. It would have been obvious to one of ordinary skill in the 
art to have made such modifications because Tomm directly teaches that the x and y 
coordinates of each box are stored in a list in memory in column 6, lines 41-45, and 
Alexander teaches of a similar list of storing the z value of each box in column 24, lines 
53-58. Alexander also teaches in column 24, lines 45-48, that "the Z axis is normal to 
the two-dimensional display screen. The Z order identifies the order of the displayed 
dialog boxes as they are layered on the display screen". Tomm shows of a desire for 
layering their boxes as may be seen in their teaching of column 6, lines 1-4, which 
teaches that their data tree contains "enclosure relationships", which is clear to one of 
ordinary skill in the art to be a form of overlay. This interpretation is further reinforced in 
Tomm's showing of Fig. 2B, where Tomm shows three different boxes (22, 23, and 25) 
"overlaying" a larger box (21). Still further, Tomm and Alexander are analogous art in 
the field of dialogue boxes. 

Response to Arguments 

26. Applicant's arguments filed 6/6/2008 have been fully considered but they are not 
persuasive. Applicant's arguments shall be addressed in the order in which they were 
presented. 
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27. In response to applicant's argument that there is no suggestion to combine the 
references, the examiner recognizes that obviousness can only be established by 
combining or modifying the teachings of the prior art to produce the claimed invention 
where there is some teaching, suggestion, or motivation to do so found either in the 
references themselves or in the knowledge generally available to one of ordinary skill in 
the art. See In re Fine, 837 F.2d 1071 , 5 USPQ2d 1596 (Fed. Cir. 1988)and In re 
Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992). In this case, motivation may 
be found within the teachings of Tomm and Tran. Firstly, Tomm states in their 
background of their invention in column 1, lines 41-57, that it was well known in the art 
that drawing programs (such as Tran's method) were used to create GUIs. Applicant 
argues on page 8 of their amendment that Tomm teaches away from such a 
modification. However, further motivation may be found in the format in which either 
system handles data. Tomm teaches of a method of utilizing a text file to generate a 
GUI, where the text file is a "two dimensional pictorial representation" of the GUI to be 
created. Tran teaches in column 7, line 66 - column 8, line 1 , that drawn data is 
converted into text data. Hence, Tran uses hand-drawn data to create text data and 
Tomm utilizes text data to create a GUI, thus making the two arts compatible with each 
other. 



28. Applicant next argues in pages 8-9, with regard to claims 1-20, that Alexander 
fails to disclose that the child dialog boxes are not annotated nor that a stacking order 
may be determined based on a data type and that neither Tomm nor Tran teaches the 
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amended limitation of "said hierarchy reflecting a stacking order based on at least one 
of an annotation to a sketched version of an object, a containment of un-annotated 
objects within an object and data types of un-annotated objects, wherein said 
annotation indicates a priority for the annotated object and said un-annotated objects 
are recognized to obtain a priory of the containing object; and wherein said stacking 
order determines to which of two objects has priority to overlay the other of the two in 
said GUr. 

The examiner respectfully disagrees. 

Tomm does in fact teach "said hierarchy reflecting a stacking order based on a 
containment of un-annotated objects within an object and data types of un-annotated 
objects" in their teaching of column 4, lines 43-45, where they teach essentially that 
hierarchy is determined based upon containment of objects. With respect to applicant's 
use of the term "un-annotated", the examiner interprets this annotation to be that of 
priority level such as those shown in their Fig. 4, numbers 410, 422, and 424, and is 
described in their specification at page 6, lines 1-5, where they specifically define these 
terms as a "list version" which indicates priority to overlay. As Tomm does not teach of 
such a priority specification, their objects are considered to be "un-annotated". 

Conclusion 

29. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure: 
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-Geaghan et al. (US 5,790,114), electronic whiteboard with multifunctional user 
interface. 

-Blume, (US 2004/0056900), system and method for window priority rendering. 
-Vienneau et al. (US 2005/0172242), generating a user interface. 
-Wakai (US 2007/0052685), information processing apparatus and GUI 
component display method for performing display operation on document data. 

30. Applicant's amendment necessitated the new ground(s) of rejection presented in 
this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP 

§ 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 
CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not 
mailed until after the end of the THREE-MONTH shortened statutory period, then the 
shortened statutory period will expire on the date the advisory action is mailed, and any 
extension fee pursuant to 37 CFR 1 .136(a) will be calculated from the mailing date of 
the advisory action. In no event, however, will the statutory period for reply expire later 
than SIX MONTHS from the date of this final action. 

31 . Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to GREGORY A. DISTEFANO whose telephone number is 
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(571)270-1644. The examiner can normally be reached on Monday through Friday, 9 
a.m. - 5 p.m.. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Doug Hutton can be reached on 571-272-4137. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

/GREGORY A DISTEFANO/ 
Examiner, Art Unit 2176 
8/30/2008 

I (Doug Hutton/ 
Doug Hutton 
Supervisory Primary Examiner 
Technology Center 2100 



